Method to generate additional level of detail when zooming in on an image

ABSTRACT

A viewer may zoom in on an image to see a portion of the image. The image may be analyzed to determine if zoom enhancement is necessary. The zoomed region may be matched to a replacement texture. The replacement texture may be used to enhance the image by replacing some or all of the image data.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application claims priority to, and is a continuation of,U.S. patent application Ser. No. 15/818,158, filed on Nov. 20, 2017,entitled Method to Generate Additional Level of Detail When Zooming inon an Image, which is hereby incorporated by reference in its entirety.

BACKGROUND

Improving image quality has been a goal of electronic image reproductionsince the dawn of black-and-white television to today's modern highdefinition flat-screen displays. Similarly, early computers presentedcrude images on small, monochrome displays and have evolved to beingcapable of displaying millions of pixels describing three-dimensionalrendered scenes. Technologies such as Virtual Reality (VR), AugmentedReality (AR), and Mixed Reality (MR) continue to push the boundaries ofrealism in image reproduction. These technologies and others have alsoenabled more interactivity with reproduced images than ever before. Thegoal of these and related technologies is to reproduce images atsufficient quality to convey a realistic image. However, zooming in onan image may yield insufficient detail and provide a poor userexperience. There remains an ever-present need to improve electronicstill and moving picture reproduction quality to achieve this goal ofreproducing realistic images.

SUMMARY

The following summary is for illustrative purposes only, and is notintended to limit or constrain the detailed description. The followingsummary merely presents various described aspects in a simplified formas a prelude to the more detailed description provided below.

A viewer may desire to zoom in on an image to see a portion of the imagein greater detail. For example, zooming in on a two-dimensional rasterimage displayed on a television screen may involve selecting a portionof the image and enlarging it to fill the entire television screen. Someexample types of images may include two-dimensional raster images,two-dimensional vector images, two-dimensional raster video sequences,two-dimensional vector video sequences, 360-degree video sequences,360-degree raster images, augmented reality (AR) images or displays,three-dimensional rendered images, virtual reality (VR) images ordisplays, mixed reality (MR) images or displays, or the like, as well asany combination thereof.

Some aspects of the present disclosure disclose a method to generateadditional levels of detail when zooming in on an image. First, andaccording to some implementations, a determination may be made regardingwhether it is necessary to enhance a selected zoom region of a sourceimage. The image quality of the zoom region may be assessed and comparedagainst various standards and measurements to determine if zoomenhancement is necessary. Next, the image content of the zoom region maybe characterized. Then, a texture database may be searched to find asubstitute or replacement texture that substantially matches the imagecontent of the zoom region. The texture database may contain rastertextures which may be stored as raster graphics or images, for example.Raster graphics describe images by describing a plurality of pixels,individual points of image data, which represent an image or scene. Anexample of a raster graphic or image format which may be used to storeraster textures is a bitmap image format. The texture database may alsocontain vector textures which may be stored as vector graphics orimages. Vector graphics describe an image by mathematical modelsdescribing lines, colors, gradients, and other such parameters whichcomprise the image. An example of a vector image format which may beused to store vector graphics or images is the Scalable Vector Graphics(SVG) format. Any other suitable image format for raster or vectortextures maybe used, including lossy or lossless image formats. Once asubstitute or replacement texture that substantially matches the imagecontent of the zoom region is identified, then the identifiedreplacement texture or textures may then be combined with the sourceimage data of the zoom region to produce an enhanced zoom region.

In other aspects of the present disclosure, a source image may becomprised of multiple segments. The source image may be segmented intomultiple segments of continuous texture character prior to zoomenhancement. Then, each segment may be individually enhanced andrejoined to re-create the original image region at a higher level ofzoom.

Some aspects of the present disclosure relate to source images that arethree-dimensional rendered scenes. The zoom enhancement method may beapplied to individual object textures in a three-dimensional renderedscene. In other aspects of the present disclosure, textures may bederived from the source image for use as replacement textures. Texturesmay also be algorithmically and/or dynamically generated rather thanretrieved from a database.

The summary here is not an exhaustive listing of the novel featuresdescribed herein, and is not limiting of the claims. These and otherfeatures are described in greater detail below.

BRIEF DESCRIPTION OF THE DRAWINGS

These and other features, aspects, and advantages of the presentdisclosure will become better understood with regard to the followingdescription, claims, and drawings. The present disclosure is illustratedby way of example, and is not limited by, the accompanying figures inwhich like numerals indicate similar elements.

FIG. 1 shows an example communication network on which many of thevarious features described herein may be implemented.

FIG. 2 shows an example computing device that may be used to implementany of the methods described herein.

FIG. 3A is an example flow diagram of a method to generate additionallevel of detail when zooming in on an image.

FIG. 3B illustrates the relationship between a source image and a zoomregion.

FIGS. 4A and 4B illustrate an incremental approach to using substitutetextures to enhance a zoomed image.

FIG. 5 illustrates an example image comprising two regions.

FIG. 6 illustrates an example flow diagram of a method to generateadditional level of detail when zooming in on an image comprisingmultiple segments.

FIGS. 7A and 7B illustrate a mipmap data structure.

FIG. 8 illustrates an example flow diagram of a method to generateadditional level of detail when zooming in on an image comprisingmultiple segments.

FIG. 9 illustrates an example flow diagram of a method to generateadditional level of detail when zooming in on an image comprisingmultiple frames and multiple segments.

FIG. 10 is an example flow diagram of a method to generate additionallevel of detail when zooming in on an image using generated replacementtextures.

DETAILED DESCRIPTION

In the following description of various illustrative examples, referenceis made to the accompanying drawings, which form a part hereof, and inwhich is shown, by way of illustration, various examples in whichaspects of the disclosure may be practiced. It is to be understood thatother examples may be utilized and structural or functionalmodifications may be made, without departing from the scope of thepresent disclosure. In the following description of various illustrativeexamples, reference is made to the accompanying drawings, which form apart hereof, and in which is shown, by way of illustration, variousexamples in which aspects of the disclosure may be practiced. It is tobe understood that other examples may be utilized and structural orfunctional modifications may be made, without departing from the scopeof the present disclosure.

FIG. 1 shows an example communication network 100 on which many of thevarious features described herein may be implemented. The network 100may be any type of information distribution network, such as satellite,telephone, cellular, wireless, etc. The network 100 may be an opticalfiber network, a coaxial cable network, or a hybrid fiber/coaxdistribution network. Such networks 100 use a series of interconnectedcommunication links 101, such as coaxial cables, optical fibers, orwireless links to connect multiple premises 102, such as businesses,homes, or user dwellings to a local office 103 or headend. The localoffice 103 may transmit downstream information signals onto the links101 and the premises 102 may have receivers used to receive and toprocess those signals.

There may be one link 101 originating from the local office 103, and itmay be split a number of times to distribute the signal to variouspremises 102 in the vicinity, which may be many miles, of the localoffice 103. The links 101 may include components such as splitters,filters, amplifiers, etc., to help convey the signal clearly, but ingeneral each split introduces a bit of signal degradation. Portions ofthe links 101 may also be implemented with fiber-optic cable, whileother portions may be implemented with coaxial cable, other lines, orwireless communication paths.

The local office 103 may include an interface 104, such as a terminationsystem (TS). For example, the interface 104 may be a cable modemtermination system (CMTS), which may be a computing device configured tomanage communications between devices on the network of the links 101and backend devices such as servers 105-07. The interface 104 may be asspecified in a standard, such as the Data Over Cable Service InterfaceSpecification (DOCSIS) standard, published by Cable TelevisionLaboratories, Inc. (a.k.a. CableLabs), or it may be a similar ormodified interface. The interface 104 may be configured to place data onone or more downstream frequencies to be received by modems at thevarious premises 102, and to receive upstream communications from thosemodems on one or more upstream frequencies.

The local office 103 may also include one or more network interfaces108, which can permit the local office 103 to communicate with variousother external networks 109. These networks 109 may include, forexample, networks of Internet devices, telephone networks, cellulartelephone networks, fiber optic networks, local wireless networks, suchas a WiMAX network, satellite networks, or any other desired network.These networks 109 may transmit content to the local office 103 via aplurality of variable size, fixed duration video fragments. The networkinterface 108 may include the corresponding circuitry needed tocommunicate on the external networks 109, and to other devices on thenetwork such as a cellular telephone network and its corresponding cellphones.

As noted above, the local office 103 may include a variety of servers105-07 that may be configured to perform various functions. The localoffice 103 may include a push notification server 105. The pushnotification server 105 may generate push notifications to deliver dataor commands to the various premises 102 in the network or to the devicesin the premises 102 that are configured to detect such notifications.The local office 103 may also include one or more content servers 106.The content servers 106 may be one or more computing devices that areconfigured to provide content to users at their premises. This contentmay be, for example, video content such as video on demand movies ortelevision programs, songs, text listings, or other types of content.The content server 106 may include software to validate user identitiesand entitlements, to locate, retrieve and receive requested content, toencrypt the content, and to initiate delivery by streaming of thecontent to the requesting user or device. The content may comprise aplurality of fixed size, variable duration video fragments. The localoffice 103 may include a load balancer (not illustrated) to routeservice requests to one of the content servers 106. The load balancermight route the service requests based on utilization or availability ofeach of the content servers 106.

The local office 103 may also include one or more application servers107. An application server 107 may be a computing device configured tooffer any desired service, and may run various languages and operatingsystems, such as servlets and JSP pages running on Tomcat/MySQL, OSX,BSD, Ubuntu, Red Hat, HTML5, JavaScript, AJAX, or COMET. The applicationserver 107 may be responsible for collecting television program listingsinformation and generating a data download for electronic program guidelistings. In some aspects of the disclosure, the application server 107may be responsible for monitoring user viewing habits and collectingthat information for use in selecting advertisements. The applicationserver 107 may be responsible for formatting and insertingadvertisements in a video stream being transmitted to the premises 102.Although shown separately, one of ordinary skill in the art willappreciate that the push server 105, the content server 106 and theapplication server 107, may be combined. Further, here the push server105, content server 106, and the application server 107 are showngenerally, and it will be understood that they may each contain memorystoring computer executable instructions to cause a processor to performsteps described herein or memory for storing data.

An example premise 102 a, such as a home, may include an interface 120.The interface 120 can include any communication circuitry needed toallow a device to communicate on one or more links 101 with otherdevices in the network. For example, the interface 120 may include amodem 110, which may include transmitters and receivers used tocommunicate on the links 101 and with the local office 103. The modem110 may be, for example, a coaxial cable modem, for coaxial cable links101, a fiber interface node, for fiber optic links 101, a twisted-pairtelephone modem, a cellular telephone transceiver, a satellitetransceiver, a local Wi-Fi router or access point, or any other desiredmodem device. Also, although only one modem is shown in FIG. 1, aplurality of modems operating in parallel may be implemented within theinterface 120. Further, the interface 120 may include a gatewayinterface device 111. The modem 110 may be connected to, or be a partof, the gateway interface device 111. The gateway interface device 111may be a computing device that communicates with the modem 110 to allowone or more other devices in the premises 102 a, to communicate with thelocal office 103 and other devices beyond the local office 103. Thegateway interface device 111 may be a set top box 113 (STB), digitalvideo recorder (DVR), computer server, or any other desired computingdevice. The gateway interface device 111 may also include local networkinterfaces to provide communication signals to requesting entities ordevices in the premises 102 a, such as display devices 112, for example,televisions, additional STBs 113 or DVRs, personal computers 114, laptopcomputers 115, wireless devices 116 such as wireless routers, wirelesslaptops, notebooks, tablets, netbooks, or smart phones, cordless phones,for example, Digital Enhanced Cordless Telephone-DECT phones, mobilephones, mobile televisions, personal digital assistants (PDA), landlinephones 117, which may be Voice over Internet Protocol (VoIP) phones, andany other desired devices. Examples of the local network interfacesinclude Multimedia Over Coax Alliance (MoCA) interfaces, Ethernetinterfaces, universal serial bus (USB) interfaces, wireless interfacessuch as IEEE 802.11 or IEEE 802.15, analog twisted pair interfaces,Bluetooth interfaces, and others.

The gateway interface device 111 or a display device 112 may be used toview video content delivered from the content server 106. Additionally,the gateway interface device 111 or a display device 112 may be used toschedule recordings of the video content or to display a program listingindicating start and end times for video content.

FIG. 2 shows an example computing device that may be used to implementany of the methods described herein. A computing device 200 may includeone or more processors 201, which may execute instructions of a computerprogram to perform any of the features described herein. Theinstructions may be stored in any type of computer-readable medium ormemory, to configure the operation of the processor 201. For example,instructions may be stored in a read-only memory (ROM) 202, a randomaccess memory (RAM) 203, a removable media 204, such as a UniversalSerial Bus (USB) drive, a compact disk (CD) or a digital versatile disk(DVD), a floppy disk drive, or any other desired storage medium. Thestorage medium may comprise a plurality of sectors, wherein a size ofeach sector of the plurality of sectors is approximately a multiple of asubstantially fixed fragment size. Instructions may also be stored in anattached, or internal, hard drive 205. The computing device 200 mayinclude one or more output devices, such as a display 206, for example,an external television, and may include one or more output devicecontrollers 207, such as a video processor. There may also be one ormore user input devices 208, such as a remote control, keyboard, mouse,touch screen, microphone, etc. The computing device 200 may also includeone or more network interfaces, such as a network input/output (I/O)circuit 209, for example, a network card, to communicate with anexternal network 210. The network I/O circuit 209 may be a wiredinterface, a wireless interface, or a combination of the two. Thenetwork I/O circuit 209 may include a modem, such as a cable modem, andthe external network 210 may include the communication links 101discussed above, the external network 109, an in-home network, aprovider's wireless, coaxial, fiber, or hybrid fiber/coaxialdistribution system, such as a DOCSIS network, or any other desirednetwork.

FIG. 2 shows a hardware configuration of the device 200, but it shouldbe understood that some or all of the illustrated components may beimplemented as software. Modifications may be made to add, to remove, tocombine, or to divide components of the computing device 200 as desired.Additionally, the components illustrated may be implemented using basiccomputing devices and components, and the same components (e.g., aprocessor 201, a ROM storage 202, a display 206, etc.) may be used toimplement any of the other computing devices and components describedherein. For example, the various components herein may be implementedusing computing devices having components such as a processor executingcomputer-executable instructions stored on a computer-readable medium,as illustrated in FIG. 2. Some or all of the entities described hereinmay be software based, and may co-exist in a common physical platform.For example, a requesting entity can be a separate software process andprogram from a dependent entity, both of which may be executed assoftware on a common computing device.

One or more aspects of the disclosure may be embodied in acomputer-usable data or computer-executable instructions, such as in oneor more program modules, executed by one or more computers or otherdevices. Generally, program modules include routines, programs, objects,components, data structures, etc. that perform particular tasks orimplement particular abstract data types if executed by a processor in acomputer or other data processing device. The computer executableinstructions may be stored on one or more computer readable media suchas a hard disk, an optical disk, a removable storage media, a solidstate memory, a RAM, etc. The functionality of the program modules maybe combined or distributed as desired. In addition, the functionalitymay be embodied in whole or in part in firmware or hardware equivalentssuch as integrated circuits, field programmable gate arrays (FPGA), andthe like. Particular data structures may be used to more effectivelyimplement one or more aspects of the disclosure, and such datastructures are contemplated within the scope of computer executableinstructions and computer-usable data described herein.

According to some aspects described herein, a viewer may desire to zoomin on an image to see a portion of the image in greater detail. Someexample types of images may include two-dimensional raster images,two-dimensional vector images, two-dimensional raster video sequences,two-dimensional vector video sequences, a frame of a two-dimensionalraster video sequence, a frame of a two-dimensional vector videosequence, two-dimensional raster moving images, two-dimensional vectormoving images, 360-degree video sequences, 360-degree raster images, aframe of a 360-degree video sequence, augmented reality (AR) images ordisplays, three-dimensional rendered images, three-dimensional renderedscenes, virtual reality (VR) images or displays, mixed reality (MR)images or displays, or the like, as well as any combination thereof. Inan example, an image may be displayed on display device 112 in a premise102.

For example, zooming in on a two-dimensional raster image displayed on atelevision screen may involve selecting a portion of the image andenlarging it to fill the entire television screen. A zoom region may beindicated, in some examples, by indicating a locus of zoom and a zoomlevel. A zoom region may also be indicated, in some examples, byindicating the boundaries or at least some boundaries of a zoom regionon a display. Touch-screen displays such as tablet computers may presentintuitive interfaces for viewers to indicate a desire to zoom in on animage. With these interactive displays, a user may indicate a zoomregion by a two-finger gesture such as a pinch gesture. In interactivedisplay devices such as a VR headset, a viewer may move closer to anobject in a virtual space displayed in the VR image to indicate a zoomregion. As the viewer moves closer to the object, the object takes upmore of the field of view of the VR headset. In these ways and more,images are becoming increasingly interactive with consumers and viewersdesiring to interact with images during viewing.

Image storage and communication mechanisms such as those implemented innetwork 100 generally have limitations on transmission bandwidth,storage capacity, and/or processing capabilities that limit how muchinformation can be used to describe an image. For example, a highdefinition television image may be encoded at a 1920×1080 resolution.Image parameters may be selected to produce an acceptable image qualityfor a certain display or output device while making trade-offs toconserve bandwidth, storage capacity, and/or processing power. Forexample, the high definition television image may be encoded at aspecified resolution and bitrate that is supported by the displaydevice, processing circuitry, and allotted broadcast bandwidth thatproduces the desired quality image for the intended display.

Because of these limitations and trade-offs in image storage andtransmission, zooming in on an image region decreases the amount of dataavailable to describe the field of view of a given display device. Thatis, zooming in on a portion of an image typically decreases the amountof the source image data that can be used to fill the same field ofview. By disregarding a portion of the source image data, zooming in onan image may be detrimental to image quality if insufficient sourceimage data is available to provide an acceptable quality image at thatzoom level.

FIG. 3A illustrates a method to generate additional levels of detailwhen zooming in on an image. The method to generate additional levels ofdetails when zooming illustrated in FIGS. 3A and 3B may increase theperceived quality of zoomed images beyond what is available from thesource image. A substitute image portion, in the form of a texture, maybe retrieved from a database and used to enhance or even replaceentirely the zoom region. The result is a hybrid or entirely syntheticimage in that at least a portion of the resultant image is derived fromthe texture database and is not necessarily present in the source image.

The method of FIGS. 3A and 3B may be implemented at various points innetwork 100. For example, a method to generate additional levels ofdetail when zooming in on an image may be implemented on set top box 113located in a premise 102 such as example premise 102 a which is a home.Similarly, other devices within premise 102 may implement the method togenerate additional levels of detail when zooming in on an image, suchas but not limited to gateway interface device 111, display devices 112,STBs 113 or DVRs, personal computers 114, laptop computers 115, andwireless devices 116. In some embodiments, the method to generateadditional levels of detail when zooming in on an image may beimplemented on devices external to premises 102, such as in local office103. For example, the method to generate additional levels of detailwhen zooming in on an image may be implemented using interface 104,content servers 106, and/or application servers 107. In someembodiments, various aspects or steps of the method to generateadditional levels of detail when zooming in on an image may beimplemented on multiple devices located remotely from one-another. Forexample, some aspects or steps may be implemented on devices located inlocal office 103 while other steps or aspects may be implemented ondevices located in premise 102.

The inputs to the method illustrated in FIGS. 3A and 3B may includesource image 313 and zoom region 317. The relationship between sourceimage 313 and zoom region 317 is illustrated in FIG. 3B. As illustratedin FIG. 3B, the source image 313 may be an image at its original or‘native’ resolution and size, filling the field of view of a displaydevice. In the example illustrated in FIG. 3B, the source image 313 is arectangular image such as a video image. In other examples, the sourceimage 313 may be a three-dimensional scene rendered for AR or VRpurposes or a 360-degree video signal, among other examples. The zoomregion 317 may be the portion of the source image that a viewer hasindicated a desire to enlarge to the same size as source image 313, aspart of a zoom operation. For example, a user may zoom in on atwo-dimensional video signal by locating a cursor on the display andpressing a “+” button on a remote to zoom into that portion of thesource image 313. In another example, a user may use AR or VR controlinputs to move in a virtual space closer to an object or a portion of anAR or VR scene to indicate a desire to zoom in on that portion of thesource image. FIG. 3B illustrates the zoom level by comparison of zoomregion 317 a as a portion of source image 313 and enlarged zoom region317 b that has been enlarged to fill the field of view of the displaydevice. Both zoom regions 317 a and 317 b describe the same portion ofsource image 313, but zoom region 317 b has been enlarged to the samesize and dimension as source image 313.

The zoom level may be proportional to the ratio of the size of theoriginal size zoom region 317 a to the enlarged zoom region 317 b. Thezoom level may be expressed as a multiplier value such as 2× or 3× zoom.Alternatively, the zoom level may be expressed as a focal lengthdistance. Using focal length to describe zoom level may be common inphotography where a shorter focal length corresponds to a low zoomlevel, and a longer focal length corresponds to a higher zoom level. Inembodiments where the image comprises a three-dimensional renderedscene, the zoom level may correspond to a virtual distance in thethree-dimensional rendered scene between a virtual camera and a virtualobject in the three-dimensional rendered scene.

Zoom level may be controlled by a user viewing image content, bymultimedia equipment at the consumption endpoint, and/or by a contentdistributor, according to some implementations. For example, a userand/or viewer may control zoom levels to increase detail on a subsectionof an image during playback. Multimedia equipment at the consumptionendpoint may include equipment such as a television display, a mediaplayback unit, a media broadcast receiving unit, or the like. Forexample, multimedia equipment may operate at a level of zoom to adaptimage content intended for one aspect ratio to another aspect ratio.Similarly, a media broadcaster may zoom content prior to broadcast forsimilar reasons.

Returning to FIG. 3A, at step 303 a determination may be made whether itis necessary to enhance the zoom region 317 of source image 313. Zoomenhancement may be necessary when a zoom level would produceunacceptable image quality. Unacceptable image quality may be judgedrelative to the subjective experience of a viewer. For example, onesubjective description of unacceptable image quality is “pixilation,”which describes the phenomenon where individual source image pixelsbecome visible to a viewer, decreasing the realism of the perceivedimage. When striving for realistic image reproduction, pixilation may beundesirable.

Quality Assessment

Various measures of image quality may be assessed to establish when zoomenhancement is necessary. Any characteristic of the zoom region may beevaluated to determine to enhance the zoom of the zoom region. One suchmeasure of image quality may be based on the spatial frequency contentof the zoomed region. The spatial frequency of an image refers to thetwo-dimensional distribution of image intensity or brightness overdistance. Spatial frequency content of an image may be determined bytwo-dimensional Fourier decomposition. The result of Fourierdecomposition of a two-dimensional signal such as an image is a weightedsum of two-dimensional basis functions corresponding to discretefrequencies. As such, the total spatial frequency content of an imagemay be characterized by evaluating the weights and frequencies of theconstituent basis functions of an image. In images, high-frequencycontent may be associated with high subjective detail. An image regionwith substantial high-frequency content may produce pixilation at alower level of zoom than an image region with mostly low-frequencycontent. For example, in an image region comprising a solid color bluesky with no fine details, i.e., primarily low-frequency content, a highzoom level may not necessarily result in loss of detail and thus may notrequire zoom enhancement. Conversely, an image region comprising of, forexample, a chain-link fence may contain more high-frequency contentcorresponding to the repeating fence links against a background. Thisimage region may deteriorate quickly as zoom levels increase because ofthe detail of the chain-link fence. The image region of a blue sky maybe zoomed in farther than that of the chain-link fence before becomingsubjectively pixelated.

The spatial frequency content of the zoomed region may be analyzed todetermine if zoom enhancement is necessary. The frequency content of azoom region may be determined by analyzing a frequency transform of theregion. For example, the Discrete Cosine Transform (DCT) frequencytransform may be used. In the DCT, an image region is described as a sumof a plurality of weighted basis functions, each representing adifferent visual frequency in the image region. Each basis functionrepresents a two dimensional cosine wave in the image region. Byevaluating the relative weights of the DCT basis functions, thefrequency content of an image region may be characterized. For example,the weights of the DCT basis functions may be visualized as a histogramand divided into three portions. If the lowest-frequency portion has thehighest cumulative weighting, the image region may be primarilylow-frequency. Similarly, if the highest-frequency portion has thehighest cumulative weighting, the image region may be primarilyhigh-frequency. And if the middle portion has the greatest cumulativeweighting, the image region may be primarily medium-frequency content.

Another measure of image quality may rely on a priori knowledge of thesource image. For example, a threshold zoom level or zoom ratio may beset between acceptable zoom and unacceptable zoom levels based onknowledge of the content. This predetermined zoom threshold may bestored and applied uniformly to all display content, and/or stored inmetadata associated with a piece of content. The a priori or pre-setzoom levels may be stored in metadata or set by a user preference, forexample. Zoom levels beyond the threshold may be determined to requirezoom enhancement, and zoom levels below the threshold may be determinedto not require zoom enhancement.

Yet another measure of image quality may be the number or percentage ofsource pixels in the zoom region. For example, a set number of pixelsmay be established so that any zoom region with fewer pixels than thethreshold level may be determined to require zoom enhancement. Thethreshold may depend on the display device and other factors such as thecompression applied to the image. The threshold may be expressed as anabsolute value, for example 1 megapixel (1 million pixels) or apercentage, such as 50% of the total pixels of the source image.

Texture Characterization and Database Search

Regardless of the methodology, step 303 may determine if enhanced zoomis necessary. If it is, the method may proceed to step 305. If not, themethod may stop, as no further processing may be necessary. At step 305,the zoom region may be analyzed to characterize the image or textureinformation in the zoom region. Then, based on this characterization,one or more substitute textures stored in the texture database 319 maybe chosen at step 307. And finally, at step 309 the substitute texturemay be integrated into the zoom region, producing an enhanced zoomimage.

At step 305, any suitable image or texture characterization method maybe used to characterize the zoom region. For example, one such approachis the local binary pattern (LBP) algorithm and family of relatedtechniques. In brief, the LBP approach describes an image or asubsection of an image by comparing pixels to neighboring pixels. Forexample, a pixel may be compared to its eight neighboring pixels along acircle, either clockwise or counter-clockwise. For each pixelcomparison, the center pixel may be compared to the neighboring pixeland a binary decision is made. In an example, if the center pixelbrightness is greater (i.e., lighter) than the brightness of a comparedneighboring pixel, a value of 1 is determined, and if the center pixelis less bright than the brightness of a compared neighboring pixel, avalue of 0 is determined. The comparisons may be concatenated into asingle 8-bit value for each pixel. This 8-bit representation maydescribe the pattern of neighboring pixels for each pixel. This exampleassumes a monochrome image, but similar techniques may be applied toimages of any color depth.

Step 305 may use metadata and other a priori information about thesource image to classify or characterize the image data or textures inthe zoom region. For example, in a three-dimensional scene, an objectmay be texture mapped with a particular texture with some metadataassociated with it. In an example, a brick wall in a three-dimensionalscene may be texture-mapped with a brick texture. The texture may bestored with metadata describing that it is a brick texture. Thismetadata and other such metadata may be used to characterize or classifythe textures in the zoom region in step 305.

Another method of characterizing the image data in the zoom region mayinclude using machine learning or computer vision techniques to identifyobjects or qualities of the image in the zoom region. For example, imageclassifiers may generate descriptive text for image regions identifyingobjects contained in the image. The output of these machine learning orcomputer vision techniques may be textual metadata or some other imagedescription.

At step 307, a texture database 319 may be searched for a suitablesubstitute texture based on the characterization produced in step 305.The texture search may be based on, for example, the LBP analysis asdescribed above. Similarly, if the texture database contains metadataabout the textures stored in it, the metadata of the zoom region texturemay be used to search the texture database 319 as well. Statisticalmethods may be used to establish a ranking or hierarchy of matches inthe texture database, with the match having the greatest ranking beingselected. Regardless of methodology used, step 307 may produce anidentification of a texture in the texture database that corresponds tothe image data in the zoom region.

Zoom Enhancement

Once a substitute texture from texture database 319 is identified instep 307, according to some embodiments, the substitute texture may beused to enhance the zoom region in step 309. One method of enhancing azoomed region with a substitute texture may be to replace the zoomedregion entirely. For example, in the chain-link fence example above, asubstitute texture of a chain-link fence may be substituted for at leasta portion of the original image at a high level of zoom. Anycharacteristic of the substitute texture may be altered to better matcha characteristic of the source image before substitution. For example,the color or chrominance of the texture from the texture database may bealtered to match the source image region being replaced. The replacementtexture may be scaled, rotated, skewed, or otherwise dimensionallytransformed to match one or more dimensions of the zoom region better.Other such parameters of the substitute texture may be modified oraltered to match the source image region such as brightness, luminance,opacity, hue, saturation, tint, shade, or other image parameters.

The substitute texture may be merged with the original zoom region imageto produce a blended effect. The substitute texture may be overlaid theoriginal zoom region and an opacity applied to produce a blended ormerged image. In some embodiments, only a portion of the imageinformation of the substitute texture may be used. For example, thechrominance information of the substitute texture may be overlaid on theenlarged zoom region image to produce a blended image. In this way, thesubstitute texture may partially replace a portion of image informationwhile retaining other portions of image information. Examples of theportions of image information that may be substituted independently ofothers include but are not limited to luminance, chrominance,brightness, opacity, hue, saturation, tine, share, or other imageparameters. The result may be a zoomed region that incorporates at leastsome image information from the texture database 319 to enhance the zoomregion 317, producing a more visually pleasing image.

Progressive Substitution

The various techniques for integrating a substitute texture with asource image may be used at various levels of zoom to produce a smoothzoom transition from low zoom level to high zoom level. FIGS. 4A and 4Billustrate an incremental approach to using substitute textures toenhance a zoomed image, according to some embodiments. FIG. 4Aillustrates a relationship between zoom level and subjective quality ina zoomed image. As an operator or viewer zooms into an image or scene,the perceived level of detail may decrease. For example, zooming in froma low level of zoom to a higher level of zoom on a raster twodimensional image may enlarge a subset of an image to the size of theoriginal image, using less of the original raster image to fill the samedisplay area. Zooming in on a three-dimensional scene with rastertextures on three-dimensional objects may have a similar effect. Thegreater the zoom level, the fewer original pixels that may be availableto fill the same display area as before.

As the viewer zooms in further and further, the perceived image qualitymay decrease. Three ranges of zoom may be distinguished, according tosome implementations. The first is the high quality zoom range 401. Inthis zoom range, the original image may contain sufficient detail todisplay a zoomed image with sufficient detail and may not requirefurther processing. For reference, this region may correspond to the‘No’ branch from decision step 303 in FIG. 3A.

The third zoom range 405 describes a range of zoom level that producesvery poor quality zoomed images. Zoom regions at these levels may bereplaced entirely by textures from a texture database because theoriginal image contains insufficient detail to produce a zoomed image atthese zoom levels.

Zoom range 403 includes zoom levels that are in between the first andthird. Here, zoomed images may be of an intermediate quality andsubstitute textures may be merged, meshed, and/or overlaid over theoriginal image to produce a blended image, as described above.

FIG. 4B illustrates three approaches to enhancing a zoomed imagecorresponding to the three zoom ranges illustrated in FIG. 4A. As thezoom level increases from left to right, the enhanced zoom region imagebecomes increasingly synthetic. That is, the substitute texture is usedrather than the image data from the original source image, in theillustrated example. High quality range 401 corresponds to normal region407, medium quality range 403 corresponds to blended region 409, and lowquality range 405 corresponds to synthetic region 411.

Segmentation

A substitute texture may replace the entire zoom region or a sub-regionof the zoom region. FIG. 5 illustrates a zoomed image comprising tworegions, 501 and 502. The division between region 501 and 502 isillustrated by line 503. For example, if the top portion of a zoomregion corresponds to a cloudy sky and the bottom portion of the imagecorresponds to water, different textures may be applied to each portionwhen zooming in on the horizon between the two portions of the image.

FIG. 6 illustrates an exemplary method for processing a segmented image.First, at step 602, the image may be segmented into contiguous textureregions. Then, each region may be enhanced independently of one anotherat step 603 according to the method illustrated in FIG. 3A. Then, atstep 605, the enhanced segmented regions may be rejoined to produce acomplete zoomed image.

Any number of zoom region segments may be formed in a zoom region, andthe zoom region segments may be of any suitable size and/or shape. Anytechnique for segmenting the zoom region may be used at step 602.Examples of segmentation algorithms include but are not limited toclustering methods such as K-means algorithms, edge detectionalgorithms, histogram-based methods, region-growing methods, or othersuch segmentation techniques, algorithms, or methods. Segmentation mayproduce any number of segments to individually process, and segmentsneed not be continuous shapes.

In embodiments where textures are mapped to three-dimensional objects,different textures may be mapped to different objects in a zoom region.In these embodiments, segmentation of the zoom region may follow theboundaries between objects described by the three-dimensional models ofthose virtual objects in the three-dimensional scene.

Once zoom enhancement for each segment is completed, the segments may berejoined at step 605. Segments may be rejoined together by fading,blurring, blending, dithering, stitching, and/or otherwise merging theimage segments back to a single zoom region image. In relation to FIG.5, for example, rejoining enhanced zoom segments may be done by blendingsegments 501 and 502 along line 503.

Rendered Images

In three-dimensional rendered images, the zoom enhancement method may beapplied to individual object textures. In this way, the decision ofwhether or not zoom enhancement is necessary described in connectionwith step 303 of FIG. 3A may be made relative to individual objectswithin the three-dimensional scene. For example, in a three-dimensionalrendered scene there may be a foreground object set against a backgroundobject. Each object may have a surface texture applied to it. Thedistance from the virtual camera to each texture-mapped object may beused to determine if each object requires zoom enhancement. Here, thetexture on the foreground object may not require zoom enhancement whilethe texture on the background object does.

Therefore, the zoom enhancement may be integrated into athree-dimensional rendering pipeline. Textures may be stored forthree-dimensional rendering in a mipmap. A mipmap is a data structurewhich stores an original size image with a sequence of progressivelyreduced size versions of the same image. A mipmap data structure for atexture is illustrated in FIG. 7A. Texture image 701 is a texture at afirst resolution, for example 512×512 pixels. Texture image 702 is ascaled version of texture image 701, in this example scaled to half ofthe dimensions of 701, or 256×256 pixels for example. Similarly, textureimage 703 is a half-scaled version of 702 at 128×128 pixels, and textureimage 704 is a half-scaled version of texture image 703 at 64×64 pixels,etc. In this way, a mipmap may store successively smaller scaledversions of a texture in one data structure. At render time, athree-dimensional rendering engine may retrieve the appropriately scaledtexture directly from this data structure without having to re-scaletextures on the fly.

If the texture is enhanced according to the methods described above, thenewly created synthetic or partially synthetic texture may be stored inthe same mipmap data structure for future use. A three-dimensionalrendering engine may then be able to use the higher-resolution enhancetexture in the future without having to re-compute the enhanced zoomimage. For example, FIG. 7B illustrates an updated mipmap with a newtexture image 705. New texture image 705 may be the product of the zoomenhancement method described above and is, in this example, computed andstored at a resolution twice the size of the previously stored textureimage at 1024×1024 pixels. The next time the three-dimensional renderingengine requires a higher-resolution version of texture image 701, it maybe stored in the mipmap data structure as texture image 705 and readyfor immediate use.

Alternatively, if integration with the rendering engine is not possible,the rendered two-dimensional raster image may be processed according tothe methods above after the three-dimensional rendering process.

The Texture Database

The texture database 319 as discussed in connection with FIG. 3Acontains a library of high-resolution textures that may be used toenhance zoom images according to the method illustrated in FIG. 3A. Insome examples, the texture database 319 is implemented on one of servers105-07 located in local office 103. In some examples, the texturedatabase 319 is implemented on one of devices 110-117 located in premise102. Texture database 319 may comprise multiple databases or datasources located in local office 103, premise 102, or otherwiseaccessible on communication network 100 or via external networks 109.These textures in this database may be derived from a variety ofsources. The database may be universal and used for all displaymaterials, or tailored to the display material of the source image. Forexample, a particular piece of content such as a movie or a televisionshow may have its own texture database. The texture database may beassociated with a genre, series, episode, movie, or any other typedescriptor of visual content. In an example, the texture database may beprovided by an external source, whether it be from the content creatoror a generic texture database associated with a display device.

In an example, the texture database may be generated from the sourceimage or source material itself. For example, similar textures may beidentified in various scenes or frames of a piece of video content.Then, higher resolution samples of similar texture may be stored in atexture database for use with that same piece of content. For example,using the chain-link fence example, a first scene may be captured withthe chain-link fence in the far background, at a long focal distance. Asecond scene may include a closer view of the same or a similarchain-link fence at a closer focal distance, so a higher resolutionsample of the chain-link fence texture may be present in the same sourcematerial. This higher resolution texture sample may be extracted fromthe source material and used to enhance or replace the lower resolutiontexture in a zoom region in another scene of the same source material.

FIG. 8 illustrates an example of this process. At step 802, a sourceimage may be segmented according to the segmentation processes describedabove. At step 803, each of the segments may be characterized using thesame characterization process as described above. Each of these texturesalong with their characterization information may then be stored in atexture database at step 804. Optionally, textures may be stored only ifthey are significantly different than any already stored in the texturedatabase, as determined by the same characterization and searchingalgorithm as discussed in connection with FIG. 3A. Then, at step 805,the zoom enhancement process described above may be performed using thetexture database from step 804. In this way, the zoom enhancement of asource image may use textures derived from that same source image.

The exemplary process illustrated in FIG. 8 may also be applied to avideo image, such as a movie. The exemplary process illustrated in FIG.9 is the same as described above in connection with FIG. 8, butperformed on each frame of the video image, illustrated in step 902.Steps 903-905 are then analogous to steps 802-805 discussed above. Theresultant texture database would include texture samples from everyframe of the movie source image.

Vector Textures

The zoom enhancement examples above generally discussed in of rasterbased images. Substitute textures may also be vector-based. Rastergraphics describe images by describing a plurality of pixels, individualpoints of image data, which represent an image or scene. Vector graphicsdescribe an image by mathematical models describing lines, colors,gradients, and other such parameters which comprise the image. Vectorgraphics, because they are mathematical models, may be zoomed or scaledwithout loss of quality. Conversely, raster graphics are limited in howfar they may be zoomed by the pixel information in the image.

Vector textures may be stored in texture database 319. The texturedatabase 319 may be compiled a priori or from the source material asdescribed above regarding raster textures. Similarly, texture database319 may be general or generic, or tailored to a particular sourcematerial. Texture database 319 may comprise all raster textures, allvector textures, or a combination of raster and texture vectors.

Vector textures stored in texture database 319 may be rasterized atvarious resolutions for use as raster textures in conjunction withraster images. Rasterization is the process of deriving a raster imagefrom a vector image. For source material or processes incompatible withnative vector textures, vector textures may be pre-rasterized orrasterized on-demand to produce raster textures of an appropriateresolution and size for a given use.

Vector textures may be derived from the source image. This process maybe referred to as vectorization, raster-to-vector conversion, or imagetracing, for example. Using vectorization, a region of a raster sourceimage may be converted to a vector representation to produce a vectortexture. The resultant vector texture may then be used in one of theabove methods to enhance a zoom of the region. Any appropriatevectorization algorithm may be used to convert a raster image region toa vector representation. Vectorization may take place in apre-processing step or on-demand as needed.

EXAMPLES

In some embodiments, substitute or reference textures may be selected tocorrelate with the source image, producing a faithful representation ofthe source image. In other embodiments, substitute textures are selectedfor reasons other than producing a faithful representation of the sourceimage. For example, replacement textures may be chosen for artisticpurposes. Content producers may select seemingly unrelated zoomreplacement textures that add a new and unique artistic element to apiece of content.

For example, replacement textures may be selected for artistic purposessuch as providing hidden information on closer zoom levels that is not apart of the source image. For example, relevant information to astoryline of a movie may be hidden in close-zoomed textures that are notvisible at a default zoom level, providing an additional level ofinteractivity to the content.

Textures may be selected for marketing or advertising purposes. Forexample, an advertisement may be inserted at certain levels of zoom.Advertisement textures may be available only at certain levels of zoom.An advertisement texture may be selected to blend in with thesurrounding source image or selected to stand out in contrast to thesurrounding source image. Advertisement textures may be targeted tocertain viewers based on the source image or other personal dataavailable in communication network 100 or otherwise available.

In an example, a texture may be dynamically or programmaticallygenerated rather than retrieved from a database. FIG. 10 illustrates anexemplary method to generate additional levels of details when zoomingthat does not rely on a texture database. The method illustrated in FIG.10 is analogous to that illustrated in FIG. 3A, but does not include atexture database 319. Rather, at step 1007, instead of searching atexture database such as illustrated by step 307, the method illustratedin FIG. 10 generates a replacement texture. The remaining stepsillustrates in FIG. 10 correspond to the similar steps illustrated inFIG. 3A.

Generated textures may be used for a number of effects, both artisticand otherwise. For example, a texture may be generated to give anillusion of infinite zoom levels. As a viewer zooms in further andfurther to a texture, additional levels of zoom texture areprogrammatically generated. This effect may be used to represent zoomlevels on a microscopic, atomic, or even sub-atomic level to add anartistic or scientific element to source material.

In an artistic example, zoom behavior may change dynamically from onezoom direction to another using generated textures. For example, zoomingin on an object may operate as expected and reveal a closer level ofzoom of that object, but when the user zooms out the surrounding scenemay be substituted with another scene, effectively transporting theviewer from one scene to the next via zoom interaction. Another exampleof artistic dynamic zoom is a zoom hysteresis example where zooming inone direction may reveal a different effect or texture than zooming inthe opposite direction.

In an example, the zoom region defines the total area of source image.That is, the zoom enhancement method may enlarge an entire source imageto a higher resolution. This procedure may also be referred to as imagescaling, image resizing, upsampling, and/or upscaling. In an example, avideo image containing 1920×1080 image data may be up-scaled to a higherresolution such as 3840×2160 using the zoom enhancement methodsdisclosed herein. An application may be to adapt content originallygenerated at a lower resolution for display on a high-resolutiondisplay.

Although examples are described above, the various features and stepsmay be combined, divided, omitted, rearranged, revised or augmented inany desired manner, depending on the specific outcome or application.Various alterations, modifications, and improvements will readily occurto those skilled in art. Such alterations, modifications, andimprovements as are made obvious by this disclosure are intended to bepart of this description, though not expressly stated herein, and areintended to be within the spirit and scope of the disclosure.Accordingly, the foregoing description is by way of example only, andnot limiting. This patent is limited only as defined in the followingclaims and equivalents thereto.

The invention claimed is:
 1. A method comprising: receiving an inputindicative of a zoom region for an image; determining a firstcharacteristic of image data associated with the zoom region;determining, based on the first characteristic, to enhance the zoomregion; determining, based on a second characteristic of the image data,a texture; and modifying, based on the texture, the image data.
 2. Themethod of claim 1, wherein the first characteristic is a distance from avirtual camera to a virtual object.
 3. The method of claim 1, whereinthe first characteristic is a measure of detail of the image data. 4.The method of claim 1, wherein the first characteristic is a zoom ratiobased on a dimension of the zoom region and a dimension of the image,and wherein the second characteristic is determined based on the zoomratio satisfying a zoom ratio threshold.
 5. The method of claim 1,wherein the texture comprises vector data.
 6. The method of claim 1,wherein the modifying the image data comprises replacing a portion ofthe image data with the texture.
 7. The method of claim 1, wherein themodifying the image data comprises: adjusting an opacity of the texture;and combining the image data with the texture.
 8. The method of claim 1,wherein: the input indicative of a zoom region indicates a zoomdirection; and the determining the texture is based on the zoomdirection.
 9. The method of claim 1, wherein the second characteristicis a local binary pattern.
 10. The method of claim 1, wherein: the imageis a frame of a video sequence; and determining the texture is based ona content type associated with the video sequence.
 11. A methodcomprising: determining, based on an input, a zoom region of an image;determining image data associated with the zoom region; segmenting,based on a first characteristic associated with an object, the imagedata into a plurality of segments; and modifying, based on a secondcharacteristic, a texture of each respective segment of the plurality ofsegments.
 12. The method of claim 11, wherein the modifying the texturecomprises selecting the texture from a texture database associated withthe image.
 13. The method of claim 11, wherein the modifying the texturecomprises searching a texture database for textures corresponding to thesecond characteristic.
 14. The method of claim 11, wherein the modifyingthe texture is based on a texture database containing textures derivedfrom the image.
 15. A method comprising: determining, by a computingdevice, a first texture associated with a virtual object in athree-dimensional scene; selecting, by the computing device and based onthe first texture, a second texture from a texture database; generating,based on the first texture and based on the second texture, a thirdtexture; and mapping the third texture to the virtual object.
 16. Themethod of claim 15, wherein the selecting the second texture is based ona distance from a virtual camera to the virtual object.
 17. The methodof claim 15, wherein the selecting the second texture is based onmetadata associated with the three-dimensional scene.
 18. The method ofclaim 15, further comprising: generating, based on the first texture, agenerated texture; and adding the generated texture to the texturedatabase.
 19. The method of claim 15 wherein the generating the thirdtexture is based on a scaled vector representation of the first texture.20. The method of claim 15 wherein: the selecting the second texturefrom the texture database comprises: generating, based on the firsttexture, a generated texture; and adding the generated texture to thetexture database; and the second texture comprises the generatedtexture.